Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update test_examples #3195

Merged
merged 9 commits into from
Jan 23, 2025
Merged

Update test_examples #3195

merged 9 commits into from
Jan 23, 2025

Conversation

andrey-churkin
Copy link
Contributor

Changes

Add quantization_aware_training_tensorflow_mobilenet_v2 to test scope

@andrey-churkin andrey-churkin requested a review from a team as a code owner January 17, 2025 09:45
alexsu52
alexsu52 reviewed Jan 17, 2025
View reviewed changes
Copy link
Contributor

@alexsu52 alexsu52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update test duration

nncf/tests/cross_fw/examples/.test_durations

Line 18 in 438871e

}
and provide a link with passed test_examples

@andrey-churkin
Copy link
Contributor Author

Build # 645 (NNCF/nightly/test_examples)
tests.cross_fw.examples.test_examples.test_examples[quantization_aware_training_tensorflow_mobilenet_v2] -- 25 min

@andrey-churkin
Copy link
Contributor Author

Please update test duration

nncf/tests/cross_fw/examples/.test_durations

Line 18 in 438871e

}

and provide a link with passed test_examples

@alexsu52 Done, Please review

@andrey-churkin
Copy link
Contributor Author

Build # 646 (with batch_size = 32)

@andrey-churkin
Copy link
Contributor Author

batch_size learning_rate optimizer metric (int8_top1)
16 1e-5 RMSProp 0.9673885107040405
32 1e-5 RMSprop 0.971464991569519
64 1e-5 RMSprop 0.971464991569519
128 1e-5 RMSprop 0.9712101817131042
16 1e-4 RMSprop 0.9449681639671326
32 1e-4 RMSprop 0.9518471360206604
64 1e-4 RMSprop 0.9727388620376587
128 1e-4 RMSprop 0.9678980708122253
batch_size learning_rate optimizer metric (int8_top1)
16 1e-5 Adam 0.9691720008850098
32 1e-5 Adam 0.971464991569519
64 1e-5 Adam 0.9712101817131042
128 1e-5 Adam 0.9727388620376587
16 1e-4 Adam 0.9296815395355225
32 1e-4 Adam 0.95286625623703
64 1e-4 Adam 0.9656050801277161
128 1e-4 Adam 0.9643312096595764

alexsu52
alexsu52 requested changes Jan 22, 2025
View reviewed changes
Copy link
Contributor

@alexsu52 alexsu52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test_examples build failed.
https://github.com/openvinotoolkit/nncf/actions/runs/12902524043

examples/quantization_aware_training/tensorflow/mobilenet_v2/main.py Show resolved Hide resolved
@andrey-churkin
Copy link
Contributor Author

New results:

CPU:

batch_size learning_rate optimizer metric (int8_top1)
16 1e-5 RMSProp 0.9678980708122253
32 1e-5 RMSprop 0.9719745516777039
64 1e-5 RMSprop 0.9737579822540283
128 1e-5 RMSprop 0.9704458713531494
16 1e-4 RMSprop 0.9492993354797363
32 1e-4 RMSprop 0.9582165479660034
64 1e-4 RMSprop 0.9607643485069275
128 1e-4 RMSprop 0.9699363112449646

1 GPU

batch_size learning_rate optimizer metric (int8_top1)
16 1e-5 RMSProp 0.968917191028595
32 1e-5 RMSprop 0.9729936122894287
64 1e-5 RMSprop 0.9717197418212891
128 1e-5 RMSprop 0.9709554314613342
16 1e-4 RMSprop 0.9475159049034119
32 1e-4 RMSprop 0.9615286588668823
64 1e-4 RMSprop 0.9666242003440857
128 1e-4 RMSprop 0.9681528806686401

@andrey-churkin
Copy link
Contributor Author

andrey-churkin commented Jan 23, 2025 •
edited
Loading

test_examples (GA): https://github.com/openvinotoolkit/nncf/actions/runs/12927090409 - PASSED
NNCF/nightly/test_examples: # 651 - PASSED
NNCF/nightly/windows/test-examples: # 312 - PASSED

@alexsu52 alexsu52 merged commit eab6b46 into openvinotoolkit:develop Jan 23, 2025
17 checks passed
@andrey-churkin andrey-churkin mentioned this pull request Jan 31, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexsu52 alexsu52 alexsu52 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@andrey-churkin @alexsu52